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ABSTRACT 



This study examines the role of resource allocation in naval 
command teams. The experiment is based on the Composite Warfare 
Commander - Distributed Dynamic Decisionmaking (CWC-DDD) paradigm 
and investigates the impact of uncertainty in neutral /enemy 
discriminability , team information structure, and leader's 
involvement in resource coordination. Seven four— member teams, 
consisting of military officers, used the CWC-DDD to combat a pre- 
programmed air threat to a naval battle group. Results show that 
increased average resource effectiveness and higher average 
identification confidence lead to greater final team strength. 
However, increased upward communications lead to lower final team 
strength. Functional (tactical vs. non-tactical) variations in 
military background have a significant impact on team performance. 
Teams that are predominantly tactical develop more coherent 
strategies, more effectively utilize their resources and have 
better team performance. Teams that are non-tactical develop 
limited strategies, assign lower confidence to their target 
identifications and have more upward communications. 
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INTRODUCTION 



I . 

A. CONTEXT OF THE STUDY 

Large scale computer systems of today have a tremendous 

impact on the way modern battles are fought. Air defense 

weapons can locate , identify and engage an enemy target 

automatically without human intervention. Ground surveillance 

radars can locate a target, designate it with a laser, and 

send the target description to a fire direction computer 

located miles to the rear of the front line. This fire 

direction computer analyzes the target, decides the best 

ammunition mix to use and picks the weapons to fire. The 

message to fire is sent to a firing element located miles from 

the fire direction center and the mission is fired. This 

process can be completed in seconds and can be accomplished 

without human intervention. 

Despite this explosion of information, communication and 
computer technology the human remains the focal point for 
decision making in large scale systems (Kleinman and 
Serfaty, 1989) . 

Humans bring to a system there own ideas and 

adaptability. In many cases this flexibility to adapt to 
changing situations is essential to the systems proper 
operation. These skills cannot at this time be programmed 
into a computer system. Also in many cases it is not 
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politically feasible to place complete control of our 
country's weapons systems in the hands of computers. 
Therefore, human control is an integral part of weapons 
systems employed by the United States . 

The controversy surrounding the USS Vincennes incident 
points out one of the problems that remain to be solved by 
systems designers. The Fogarty report concluded that "The 
AEGIS combat system's performance was excellent - it 
functioned as designed" (Hill, 1989) . The report went on to 
say that "mistakes made by the Vincennes CIC crew contributed 
to the commanding officers belief his vessel was in danger of 
attack" (Hill, 1989) . This conclusion was disputed by Martin 
Hill in an article in San Diego Magazine (Hill, 1989) . He 
maintains that the designers of the AEGIS system failed to 
incorporate enough human engineering in their design. Without 
addressing the fault of this incident it seems clear that 
there exists a need to better understand the process of human 
decision making as it interfaces with large information 
systems . 

The AEGIS system is only one example of a large scale 
military weapons system. These systems include large 
databases, communications networks, automatic firing systems, 
several different information gathering systems (radar, sonar, 
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observers, etc) and human decision makers. More importantly 

these systems tend to have many human decision makers . 

A characteristic feature of these systems is the presence 
of a team of human decisionmakers who may be 
geographically separated, but who must coordinate to share 
their information, resources and activities in order to 
attain their goals in what is generally a dynamic and 
uncertain task environment (Kleinman and Serfaty, 1989) . 

The study of the dynamics of this distributed decision making 

was the purpose of the research sponsored by the Office of 

Naval Research in 1985. 

B. RESEARCH BY THE OFFICE OF NAVAL RESEARCH (ONR) 

The weapons of modern warfare are both accurate and 
lethal. They can destroy cities, sink ships and wipe out 
entire formations in seconds. The ability of modern 
information systems places an enormous amount of data about 
potential targets for these weapons in the hands of tactical 
decisionmakers. Additionally the effects of these weapons can 
be felt almost immediately. Thus, although a modern commander 
has extremely destructive weapons at his disposal and massive 
amounts of information about where to use these weapons, 
employment decisions have to be made in seconds. Ship 
commanders of today have precious few seconds to identify an 
approaching aircraft and determine a course of action. 
Typically in military organizations these decisions are made 
by teams . 
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Specifically, in the case of the battle fought by naval 
ships the team is composed of the Composite Warfare 
Commander (CWC) and his immediate subordinates: the Anti-Air 
Warfare Commander, the Anti-surface-Warfare Commander, and the 
Anti-Submarine Warfare Commander. Each of the subordinates 
makes decisions and passes along information to the commander. 
He makes decisions about the overall battle and passes down 
information and instructions to his subordinates. Elaborate 
and integrated information systems have been developed to 
assist these commanders make their decisions. How commanders 
make decisions in this computer assisted environment is a 
growing concern in military organizations. 

In order to understand the dynamics of team decisionmaking 
in the Navy the Office of Naval Research has initiated the 
Distributed Tactical Decision Making (DTDM) program. (Vaughn, 
1990) The purpose of this study is to understand the problems 
of distributed command and control. The program was initiated 
to look at a wide range of methodologies . The normative - 
descriptive method and lab simulation forms the basis for the 
current research. 

A computer based paradigm designed to simulate "real 
world" Navy engagements has been developed by a research team 
at the University of Connecticut and the research corporation 
of Alphatech. (Kleinman and Song, 1990) The Composite 
Warfare Commander - Distributed Dynamic Decisionmaking (CWC— 
DDD) paradigm was the basis of an experiment conducted at the 
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Naval Postgraduate School during August 1991. The Resource 
Allocation in Naval Command Teams (RAINCOAT) experiment was 
designed to study the impact of decision making variables 
using military personnel as the decision makers . 



C. THE RAINCOAT EXPERIMENT 

The study of distributed decision making is more then a 

study of the decision making steps that individuals go through 

to make decisions . In this case the sum of the parts can be 

more then the whole or if things are going wrong the sum can 

definitely be less then the whole. Serfaty and Kleinman 

summarized the state of the art in the study of distributed 

decision making at the beginning of their study as follows: 

The scientific study of distributed human decision 
problems has long been hindered by several objective 
factors. The inherent complexity of the mathematical 
formulation and solution of decentralized problems in 
control, detection, data fusion, and resource allocation 
theories has prevented the development efficient and 
practical models that could be used as a basis to predict 
actual team performance in a distributed environment.... 
Another hampering factor is the lack of psychological or 
cognitive models of team decisionmaking behavior. . . . 
Finally the lack of comprehensive empirical data has 
prevented the development of essential scientific 
hypotheses related to team decisionmaking performance. 

(Kleinman and Serfaty, 1989) 

The recognition of these deficiencies in this field of 
study and the real world need to better understand the 
dynamics of team decisionmaking lead to the initiation of this 
study by ONR. As stared earlier, the study concentrates on 
solving the types of problems encountered in naval command 
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and control systems which involve distributed decisionmaking. 
The scenarios developed thus far use computer simulations 
modeling the composite warfare commander (CWC) doctrine 
employed by naval battle groups. However, the results of the 
study conducted so far have application in many different 
military and civilian situations involving command and control 
of distributed resources . 

The RAINCOAT experiment sought to study the dynamics of 
team decisionmaking as it was practiced by teams of military 
officers. The Composite Warfare Commander — Distributed 
Dynamic Decisionmaking (CWC - DDD) paradigm was used to 
present each four person team with an abstracted military 
situation in which they must respond to an enemy air attack. 
The teams were presented with unidentified targets which they 
must identify as enemy or neutral . Each team received at 
least four hours of training prior to recorded trials of the 
experiment . The goal of this training was to create expert 
teams with a complete understanding of the mechanics of the 
simulation. A more complete description of the experimental 
procedures used in the experiment is contained in chapter III 
of this document . 
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II. CURRENT RESEARCH 



A. DEVELOPMENT OF DDD 

The current Composite Warfare Commander - Distributed 
Dynamic Decisionmaking (CWC - DDD) paradigm is the result of 
five years of study involving the performance of model driven 
basic experimental research. The development has been a joint 
effort between the University of Connecticut and Alphatech 
Inc. working under a grant from the Office of Naval Research. 
The paradigm is designed to support empirical research and 
laboratory simulation examining distributed decision making 
issues in four-person hierarchial teams of naval commanders. 
The players plan, coordinate, and allocate resources in a 
simulated naval battle situation. 

The paradigm is implemented as a computer driven 
interactive game in which four team members attempt to defend 
a naval battle group. The paradigm can be manipulated to 
expose the decision makers to different situations of 
information processing and resource allocation under different 
organizational hierarchies and information structures. The 
decision makers are expected to manipulate limited shareable 
resources to process a variety of targets in a changing 
environment. Each decisionmaker sits at a different 
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workstation which displays the situation as seen by each 
decisionmaker. (Kleinman and Song, 1990) 

B. PAST EXPERIMENTAL RESEARCH IN TEAM DECISIONMAKING 

1 . Experimental Efforts in Hierarchial Team 

Decisionmaking 

A compilation of the results of current studies of 
team decisionmaking dynamics is contained in Table 1 . This 
table gives a brief summary of the experimental research that 
has been conducted under the sponsorship on the Office of 
Naval Research. 

2 . Summary of Experimental Results 

a. Hierarchical Information Processing (HIP) 

This experiment was an essential part of a 
coordinated study analyzing team situation assessment and 
information coordination. The experiment studied situation 
assessment in three-person teams. A team leader was forced to 
assimilate information gained from subordinates with his own 
information and opinions to determine if a contact was a 
neutral or an enemy. The study was designed to look at the 
effects of feedback (in the form of subordinates opinions and 
confidence in their opinions) on the leaders assessment of the 
situation. The experimental results supported the hypothesis 
that the feedback mechanism helped to insulate teams against 
the lose of a critical element of information. The results 
hint that with feedback team members form a mental model of 
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the other decision makers and thus are able to coordinate team 



actions with less communication. (Burton and Kleinman, 1990) 



TABLE 1 

EXPERIMENTAL RESULTS IN TEAM DECISIONMAKING 



EXPER 

IMENT 


VARIABLES 


SUBJECTS 


RESULTS 


HIP 


Information 

Feedback 


3 Person 
Teams 


Feedback insulates team 
from information losses 


HRA 


Tempo 


4 Person 
Teams 


Leaders pool resources 
and positively affect 
task division 


INCO ; 


Information 


4 Person 
Teams 


Shared opinions among 
subordinates reduces 
need for leader and 
increases effective use 

nf 4 nfnnnat" i nri 
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HITEC 


Time Stress 
Leader 
Involvement 
Commo Structure 


4 Person 
Teams 


Explicit coordination 
is most greatly 
affected by tempo 
increases 


REST 


Reward Structure 


2 Person 
Flat Team 


Differing goals in 
teams forced an 
increase in explicit 
coordination and 
increased perceived 
workload. 


CREST 


Performance 

Feedback 

Expertise 

Overlap 


2 Person 
Team 


Performance feedback 
did not greatly enhance 
team performance 


ICS 


Information 

structure 

Tempo 

Command Strategy 


4 Person 
Team 

i 


Increase in external 
load increases the 
leader role. Leader can 
suffer information 
overload 
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Jb. Hierarchical Team Resource Allocation ( HRA ) 

This experiment was the concluding experiment in a 
series of studies looking at team resource allocation, action 
selection, and team coordination. This experiment examined 
the role of a leader in resource allocation problems . The 
experimental goal was to determine if a teams performance 
increased with the introduction of a team leader and a 
hierarchical structure . The leader became the resource 
allocator with the ability to force transfers of resources 
among subordinates. The researchers concluded that the 
introduction of a leader had a positive effect on the teams 
ability to coordinate an overall pooling of resources . They 
also concluded that the team was better able to coordinate 
actions to process high value tasks . These studies (HIP and 
HRA) served as the base of information at the beginning of the 
research effort sponsored by the Office of Naval Research. 
(Miao, Luh, and Kleinman, 1990) 

c. Information Coordination in Human Teams (INCO) 

This experiment was designed to study information 
coordination in hierarchical teams . The experiment used four- 
person teams. The team leader was asked to identify a contact 
as friend or foe based on information passed to him by his 
subordinates and on his own information. The subordinates 
were asked to either identify the contact and pass this 
information to the team leader or request additional 
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information. This additional information cost the team and 
reduced their overall score. The team goal was to maximize 
that score. The experiment found that when team members 
share their opinions the team uses information more 
effectively and the leader' s involvement is much less 
critical. The researchers also found that when the 
subordinates did not share information the team performed 
better with active leader. (Kleinman and others, 1991) 

d. Hierarchial Team Coordination (HITEC) 

This experiment manipulated the time available for 
decisions, the leader's role and the communication structure. 
The experiment revealed that with a 35% increase in the game 
tempo there was a 10% drop in team performance, a 25% drop in 
the number of communication messages and a 10% drop in the 
database (task status) updates. The communication messages 
with the longest planning horizon were reduced the most. The 
tasks that required more coordination were the first to be 
dropped. (Wang and others, 1991) 

e. The Reward Structure Experiment (REST) 

This experiment measured task processing in two 
person teams under an individual reward system and under a 
team reward system. The different goals at the local levels 
forced more explicit coordination, increased the perceived 
workload, reduced the number of cooperative team actions and 
lowered the overall performance. In other words the teams 
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that sought a team goal performed better at both the 
individual and team levels. (Kleinman and others, 1991) 

f. Conflict Resolution in Teams (CREST) 

This experiment investigated conflicts in resource 

sharing in two person teams . The team was confronted with a 

series of air and subsurface targets/tasks. The team was 

required to identify and then track each of these tasks for a 

specified period of time. DM1 was required to track air tasks 

while DM2 was responsible for subsurface tasks. The results 

suggested performance feedback does not significantly effect 

team performance. Rather it reduces the degree of 

cooperative behavior in a team. The experimenters concluded: 

Feedback helps correctly calibrate the mutual models that 
decisionmakers hold of each other and that feedback is 
filtered and interpreted more so at an individual level 
then at a team level. (Nodoushani, Kleinman and Serfaty, 
1989) 

g. Information and Command Strategy (ICS) 

This experiment was designed to investigate the 
effect of different information structures and command 
strategies upon resource allocation in hierarchical teams. 
Four-person teams consisting of a leader and three 
subordinates are given limited resources to process tasks 
within a given area. The independent variables in this 
experiment were leader information, command options, and the 
external load applied. The experiment showed that if a leader 
has too much authority and not enough information they tend to 
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intrude into lower level processes. Additionally it showed 
that too much information can lead to information overload. 
(Shi, Luh and Kleinman, 1990) 

C. A VIEW OF TEAM DECISIONMAKING IN COMMAND AND CONTROL 
1 . The Command and Control Environment 

Tasks assigned to command and control elements are 
dynamic in nature; they can be diverse and completely 
unpredictable. The teams that are assembled to perform 
command and control are often separated geographically and 
into different functional responsibilities. They make 
decisions in an uncertain environment where there is not a 
easily defined correct answer. Typically there is an adversary 
who will work to confuse the decisionmakers of the team. In 
these situations the team' s tasks can be broken into two 
distinct areas. They are situation assessment and resource 
management . 

Situation assessment involves a continuous analysis 
and estimate of the current situation. It requires that the 
team control information gathering resources and place them 
in positions that will maximize their effectiveness. This 
assessment requires that the team have a complete 
understanding of the strengths and weaknesses of the resources 
available to them. In a military situation this involves the 
use of radars, detection devices, IFF devices to identify 
friendly forces, special operation forces and many other 
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intelligence gathering forces and equipment. In business this 
can involve a study of the market, surveys of potential 
buyers, research of local newspapers, or even lobbying of the 
government . 

Once this information is gathered it must be processed 
to present a clear picture of the situation. This process 
involves statistical hypothesis testing, pattern recognition 
and in many cases judgement of the decisionmakers . This step 
can be one of the most difficult to implement because huge 
amounts of data can be collected in a very short time. The 
decisionmaking team must find a way to extract the pertinent 
data and assemble it into a clear picture of the situation in 
time to affect a rapidly changing situation. 

Resource management is the control and allocation of 
an individual's resources in order to utilize them in a 
effective and efficient manner to achieve organizational 
goals . When allocating resources the team must remember that 
the situation assessment is ongoing and the situation may be 
changing. Therefore, the teams must allow for a strategic 
reserve of resources to meet new demands. Teams must use 
their judgement, experience, and obtained data to decide when 
and where to use this reserve . 

The execution of the team tasks is influenced by the 
goals that the individual members of the team perceive as 
their own and those of the team as a whole. The REST 
experiment showed that those teams that were able to integrate 
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individual goals into the team goal attained much higher 
performance levels. The leader in this case is challenged to 
find common goals for his subordinates and to define them in 
a way that everyone understands . 

2. Team Coordination - The G.R.I.T. View 

Serfaty and Entin present a model of team 
decisionmaking that defines a task as a general coordination 
problem. They define coordination as the "process of managing 
the overlap (interdependencies) among co-acting 
decisionmakers" (Kleinman and others, 1991) . Furthermore 
Serfaty and Entin maintain that this coordination is an 
essential element of the team decisionmaking process. 

Serfaty and Entin define four components of 
coordination: goal coordination (G) , resource coordination 
(R) , information coordination (I) and task/action coordination 
(T) . Goal coordination involves the team in the 
reconciliation process between individual and team goals. 
Resource coordination involves an accounting of available 
resources; when and where they will be available. Information 
coordination involves the deciphering of incoming data. It can 
also be as simple as everyone agreeing on a common set of 
symbols to depict actual items. Task/action coordination 
involves the deconflicting of action. For example, two members 
of an air defense team should not inadvertently process the 
same target at the same time . 
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The present line of research is investigating how this 
coordination is facilitated by different environments. 
Specifically, the research is looking into how coordination is 
affected by the team's communication methods, the team's 
information sharing, preplanned rules or procedures, and the 
team's hierarchial structure. 

3 . Summary of Conclusions from Current Research 
a. Team Limitations and Biases 

Strategies developed by decisionmakers have a short 
planning horizon; two to three stages of decisionmaking appear 
to be the limit. Decision strategies developed by teams tend 
to be different then strategies developed by individuals. 
Individual decisionmakers tend to believe that their judgement 
of the situation is the best and that their task is the most 
important. When possible, teams tend to overuse communication 
channels in an attempt to reduce future uncertainty. This is 
particularly true when the workload is light. Finally, teams 
tend to underestimate their ability to coordinate implicitly. 
(Wang, Serfaty, Luh and Kleinman, 1991) 

Jb. Team Adaptation to Time Stress 

The level of stress that a team encounters 
influences the their performance in different ways. As the 
external load began to increase the teams prioritized their 
tasks and eliminated those with a low priority (Miao, Luh and 
Kleinman, 1990) . At this stage the teams seemed to prefer 
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processed information to raw data. Additionally, the amount 
of communication among the team members tended to decrease 
sharply. Decisionmakers shrank their planning cycle and 
eliminated tasks that required extensive coordination. 
Although the teams were beginning to encounter problems, the 
level of performance usually remained fairly high. 

As the external load increased even further the 
teams performance decreased significantly. The teams tended 
to disintegrate into a collection of individual decisionmakers 
instead of a cohesive team. The load was spread unequally in 
order to reduce the amount of coordination required between 
members of the team. This is unfortunate for the 

decisionmaker who happens to be in the busy sector. 
Additionally, the error rate showed a significant increase as 
team performance decreased. (Wang and others, 1991) 
c. Communication Strategy 

As the tempo of the external load increased from a 
low tempo to a moderate tempo the teams explicit coordination 
and resource transfer rates tended to increase. However, as 
the tempo increased to a high rate, the explicit coordination 
dropped to a rate significantly lower then the low tempo rate. 
The researchers concluded that the high tempo forced the team 
members to behave more like individual decisionmakers then as 
a coordinated team. (Wang and others, 1991) 
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d. Team Leader 



The role of the team leader diminishes as the 
subordinates are provided with communication methods for cross 
coordination or if information sources are shared throughout 
the team. The team leader is affected by the type and amount 
of information received. The team leaders authority must be 
consistent with the information that is available. Blanket 
authority without information causes the team leader to 
intrude on the decisionmaking of his/her subordinates. 
However, too much information can quickly overload the team 
leader and render decisionmaking impossible. (Shi, Luh and 
Kleinman, 1990) Leaders at different levels tended to hold 
different perceptions of the team problem. These perceptions 
caused internal conflict in the team and forced differing 
coordination methods at the different hierarchial levels in 
the team. (Wang and others, 1991) 

4 . Research Questions 

The goal of the RAINCOAT experiment is to provide 
further information into the dynamics of team decisionmaking 
as practiced by teams of military decisionmakers. The research 
question is: what is the impact of uncertainty, differing 
information flows and different leader roles on team 
performance and team decisionmaking procedures. 
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Ill . METHODOLOGY 



A. EXPERIMENTAL DESIGN 

The purpose of this study was to examine the effects of 
certainty of target identification, the command structure, and 
the role of the leader on team performance, team strategy, and 
team coordination. A double 2x2 repeated— measures within- 
subject design was used. As Figure 1 indicates the design 
allows us to examine three main effects (U, I, and L) and two 
interaction effects (U x I and I x L) . 

1 . Independent Variables 

The independent variables for this study were (1) 
Neutral/Enemy Discriminability Uncertainty (2) Team 
Information Structure and (3) Leader's Involvement in Resource 
Coordination. Neutral/Enemy Discriminability Uncertainty (U) 
was established by having all enemy aircraft fly at either 0.4 
kft (low uncertainty) or 0.5 kft (high uncertainty) . In both 
the low and high uncertainty environments neutral aircraft 
flew at 0.7 kft . In both cases sensor noise caused readings 
to fluctuate with a standard deviation of 0.2 kft and the 
scale was truncated at two standard deviations. So, for 
example, 100% of enemy aircraft in a low uncertainty situation 
would have readings from 0.0 to 0.8 while neutral aircraft in 
the same situation would have readings from 0.3 to 1.1. These 
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Pass i ve 




Decent . Centra I i zed 

Figure 1 - Experimental Design 

numbers show that readings from 0.3 to 0.8 could be either 
enemy or neutral showing how uncertainty is built into the 
study (see Figure 2) . 

Team Information Structure (I) was termed either 
Centralized or Decentralized. The leader's information set 
always included all targets (enemy, neutral, and 
unidentified) . This independent variable pertains only to the 
subordinates' information structure. In a Decentralized 
environment the subordinates' information set includes targets 
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Enemy Neutral 




Enemy Neutral 




Figure 2 - Neutral/Enemy Discriminability Uncertainty 

in their own area of responsibility (including own overlap 
areas) only. In a Centralized environment the subordinates' 
information set includes targets in their own area of 
responsibility and any enemy identified as such in all three 
areas . 

Leader Involvement in Resource Coordination (L) was 
termed either Active or Passive. In an Active leader role, 
the leader was involved in situation assessment and enemy 
attack determination, and was allowed to take an active part 
in resource coordination and platform transfer among the 
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subordinates (including advice on transfers and forced 
transfers) . 

In a Passive leader role the leader was only involved 
in situation assessment and enemy attack determination, but 
was not allowed to take part in resource coordination and 
platform transfer among the subordinates. 

2 . Raincoat Counter-Balanced Design 

Six basic scenarios were designed, with an embedded 
two-level variable: team coordination requirement. Depending 
on the direction of the main enemy attack, either resource or 
task coordination is being stressed. The scenarios were 
randomized across teams and experimental conditions in such a 
way that every Exam was treated once by each of the six 
experimental conditions and by each scenario once. Three of 
these scenarios stressed resource coordination by placing the 
primary attack in the middle of a DM' s own area of 
responsibility (each DM was given one primary attack) . 
Resource coordination requires the DM to basically work on 
his/her own and request additional resources as required. The 
other three scenarios stressed task/action coordination by 
placing the primary attack in an overlapping sector (each 
overlap sector was given one primary attack) . Task/action 
coordination requires two adjacent DM' s to work together in 
deciding who is going to work on which target (s) . Table 2 
shows how the design was counter-balanced for this experiment. 
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TABLE 2 



COUNTER-BALANCED DESIGN 



SCENARIO 




1 


2 


3 


4 


5 


6 


Experimental 
Condition 
(U, I, L) 


11 


12 


13 


21 


22 


23 


111 


A 


B/G 


D 


F 


E 


F 


121 


E 


D 


C 


E 


D 


B/G 


211 


E 


F 


B/G 


D 


C 


A 


221 


D 


E 


A 


F 


B/G 


C 


212 


C 


A 


F 


B/G 


D 


F 


222 


B/G 


C 

1 


E 

1 


A 


F 

i 


D 

1 

1 



B . SUBJECTS 

Subjects for the experiment consisted of 27 active duty 
military officers and one civilian employee of the National 
Security Agency enrolled in the Command, Control, and 
Communications (C 3 curriculum at the Naval Postgraduate 
School, Monterey California. Five of the subjects were female 
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and 23 male. The subjects were allowed to freely divide into 
seven teams of four members each. Within each group, one 
person was picked to be the leader (DM0) and the other members 
played the roles of three subordinate decisionmakers (DM1, 
DM2, and DM3) . During the training and practice sessions the 
team members were allowed to freely change positions to get a 
feel for what was required of each position. Once the real 
experimental sessions began changing of positions was 
prohibited. The level of experience was controlled by using 
the Composite Warfare Commander - Distributed Dynamic 
Decisionmaking (CWC-DDD) experimental paradigm. None of the 
subjects had previously used the CWC-DDD simulation. 

C . APPARATUS 

In our experiment the CWC-DDD simulation software was set- 
up to run on a Sun Microsystems SPARC 1+ server with four 
SPARCstation SLC diskless workstations, each having a 
monochrome monitor and an optical mouse, all linked together 
through an ethernet . The software was written to run under 
SunOS and the Sunview operating environment. Team members 
participated in the simulation using the workstations while 
the server ran the simulation. The network and experiment 
were in a room where noise was kept to a minimum and 
distractions that might influence the outcomes were minimized. 
Each DM had adequate room and was kept from seeing the other 
DM' s monitors . 
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D . TASKS 

The tasks carried out by the subjects were conducted using 
the CWC-DDD experimental paradigm. The CWC-DDD simulation 
creates a tactical environment that replicates, with a 
considerable level of abstraction, decisionmaking problems 
encountered by commanders in naval battle groups. It was 
developed by a research team at the University of Connecticut 
for use in the CREST and ICS experiments to provide a rather 
abstracted military environment. Certain features of the CWC- 
DDD simulation were greatly simplified or even removed to 
allow us to focus on a particular aspect of the team 
decisionmaking process for purposes of the RAINCOAT (Resource 
Allocation In Naval Command Teams) experiment. 

1 . Problem Domain 

The problem domain constructed for the RAINCOAT 
experiment consisted of three parts : 



• Anti-Air Problem (Inner/Outer Air Battle) . The objective 
of the blue team is to defend a defined area around an 
aircraft carrier against an orange air threat. 

• Four-Person Blue Teams. A leader (Anti-Air Warfare 
Commander - DM0) and three subordinates (DM1, DM2, and 
DM3) responsible for three geographical sectors (see 
Figure 3) . 

• Orange Attack Scenario is pre-programmed in the simulation 
(there were six pre-programmed scenarios used for this 
experiment) . Targets are maneuvering in the theater of 
operations. However, the targets are not reactive to the 
actions of the blue team. 
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Phases 



2 . 

Each experimental session is broken down into three 
phases: Planning (PL), Situation Assessment (SA) , and Threat 
Prosecution (TP) . 

a. Planning (PL) Phase 

During the PL phase the blue team receives a 
detailed description of their own resource capabilities (how 
many and what type of platforms and subplatforms) , their 
relative emplacement in future operations, a brief 
intelligence assessment of the overall situation (whether the 
enemy is expected to attack) , and a general description of the 
probable enemy attack tactics (that the enemy will use a 
primary and diversionary attack path) . The job of the blue 
team is to prepare a coordinated defense plan based on this 
preliminary information, answer a few questionnaires, 
preposition their platforms, and wait for the start of the SA 
phase . 

Jb. Situation Assessment (SA) Phase 

During the SA phase the job of the subordinates is 
to allocate their sensors (platforms and subplatforms) and to 
take sensor readings in order to make a neutral or enemy 
assessment on a target -by-target basis, in their sector of 
responsibility. Figure 3 presents an example situational 
display for a subordinate commander showing the division of 
responsibility using pie-shaped geographical sectors for a 
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Figure 3 - Typical Screen 



typical CWC-DDD simulation. In the next step the subordinates 
report their assessment to the team leader (interpret their 
sensor readings as either enemy or neutral, what size or 
damage-inducing potential the target has, and how confident 
they are in their decision) . The job of the leader is to 
request more information on a particular target if necessary 
and once satisfied with the accuracy of all subordinates' 
reports, to look at the global situation and make an attack 
determination. He should decide if the blue forces are under 
attack, where is the main avenue of attack or threat axis (in 
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what sector) , where is the eventual secondary or diversionary 
attack, and when the blue team should start threat prosecution 
for the purpose of self-defense . 

c. Threat Prosecution (TP) Phase 

During the TP phase the job of the subordinates is 
to allocate their subplatforms to engage the threat as defined 
by the leader. Each subplatform has a radius of lethality 
where an attack can be undertaken with a certain probability 
of kill. To do their job most effectively they must transfer 
resources (subplatforms) to the heaviest attack area and under 
the responsibility of that area commander. They may have to 
cooperate on some defense missions by pooling their resources 
together against a certain threat type . The job of the team 
leader is to ensure that the main goal of defending the 
carrier is achieved. To do so, he may have to enforce 
resource transfer among subordinates, set priorities on 
threats, and coordinate the subordinate's actions. A certain 
number of strength points are taken away from the team when a 
threat penetrates the inner defense zone, the blue team 
attacks a neutral, or the blue team does not allocate the 
proper amount of resources when attacking an enemy target. 
The objective of the team is to maximize the number of 
strength points remaining at the end of a scenario. At the 
completion of the trial a subjective workload assessment is 
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completed by each member of the team and finally a post- 
engagement questionnaire is distributed to team members. 

E . EXPERIMENTAL PROCEDURE 

Initial exposure to the experiment was accomplished 
through a briefing which covered very general characteristics 
of the simulation. Scheduling and other coordination was 
accomplished during this session so that each team would be 
able to provide a minimum of two hours per week for five 
consecutive weeks to the experiment . Each team was required 
to provide a minimum of ten hours to the experiment, four 
hours for training/practice and six hours for actual data 
collection . 

1 . Training and Practice Sessions 

The first part of team training involved a walk 
through using hard copy screen dumps from an actual session. 
This method was used to explain each component of the screen 
to all members simultaneously. Different screen dumps showed 
various stages of the scenario and each window that a DM might 
see was fully explained. This portion of the first session 
lasted from 45-60 minutes. After completion of the walk 
through, each team member was seated at a station and the 
simulation was started (using training scenarios) . During 
this first (and all subsequent practice sessions) run of the 
simulation two tutors were present to show different 
techniques and answer any questions posed by the subjects. 
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This part of the session lasted 35 minutes. Finally, to 
complete the first two hours of training, a practice session 
was started and the team was allowed to play freely, asking 
questions as necessary and having particular points stressed 
by the tutors. During the training/practice sessions team 
members were allowed to freely discuss anything about the 
simulation. The second training/practice session normally 
started with a review of the previous session. The remainder 
of the session consisted of running consecutive trials until 
the two hours was completed. By this point, most teams were 
confident in their ability to continue on with the data 
collection sessions. However, one team asked for another 
training session to work on some areas with which they were 
having trouble. 

2 . Data Collection Sessions 

These sessions started out with briefings on the 
various parameters of the current session and pre-engagement 
questionnaires. During these trials no talking was allowed 
between DM' s except through communications provided in the 
simulation. Each trial lasted approximately 35 minutes; 
however, 60 minutes were allotted for pre- and post -engagement 
questionnaires, a short break between trials, and any strategy 
discussions that the team conducted during the planning phase. 
Data was collected by the network and stored for later 
analysis . 
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3 . Post Data Collection 



After completion of all data collection sessions a 
short demographic questionnaire was completed by each subject 
and team strategies were discussed. 

F . DEPENDENT VARIABLES 

The dependent variables for this experiment were collected 
in order to capture the team behavior during each experimental 
scenario. They can be divided into three categories: 
Performance, Strategy, and Coordination. 

1 . Team Performance Measures 



• Final Team Strength [%] . This is the most global measure 
of team performance. It is the one that each team 
attempts to maximize. At the start of the scenario 
(time=0) , the team is given an initial strength S 0 , 
representing some aggregate value of the defensive 
strength of the battle group. As the battle progresses, 
the team strength is being reduced by the damage caused by 
enemy forces as well as by mistakes made by the blue team, 
such as attacking a neutral task. The relative team 
strength at any time is computed as: 



S 0 -AS 
S 0 



where AS is the loss at time t . This loss is a function 
of the number of threats that penetrated the inner 
defense, the number of neutral tasks attacked by the blue 
force, and the number of enemy targets that were attacked 
with inadequate resources by blue forces . 

• Number of False Alarms [] . This reflects a Type II 
misclassif ication error (target identified as enemy when 
it is really neutral) . 
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Number of Correct Enemy Identifications []. 



Number of Misdetections [] . This reflects a Type I 
misclassif ication error (target identified as neutral 
when it is really enemy) . 



Number of Correct Neutral Identifications [] . 



Correct Enemy Classification Ratio [%] . This measure is 
computed as the ratio of correctly classified enemies 
(Low, Medium, and High damage-producing capability) to the 
total number of enemy targets that are identified as such. 



Number of Targets Unidentified [] . 



Correct Primary Attack Determination [0/1] . If the leader 
correctly identifies the axis of the primary attack P ( + /- 
1 sector), then this variable takes a value of 1, 
otherwise it is 0. 



Correct Diversionary Attack Determination [0/1]. If the 
leader correctly identifies the axis of the diversionary 
attack D ( + /- 1 sector) , then this variable takes a value 
of 1, otherwise it is 0. 



Primary/Diversionary Attack Confusion [0/1] . If the 
leader incorrectly determines the axis of the primary 
attack p on the true diversionary attack D axis ( + /- 1 
sector) , then this variable takes a value of 1, otherwise 
it is 0 . 



Diversionary/Primary Attack Confusion [0/1] . If the 
leader incorrectly determines the axis of the diversionary 
attack D on the true primary attack axis P (+/- 1 sector) , 
then this variable takes a value of 1, otherwise it is 0. 



Average Attack Score on Enemy Targets [points] . This 
score is the function of the value of the enemy target 
(damage-inducing potential) and the amount of attack 
resources used on the target. For enemy target k, the 
score is computed as 



sc k -val k *ref k , 

where VAL is the target value (Low=l-4, Medium=4-6, and 
High=6-8) , and REF is the resource effectiveness for 
target k. REF is defined in Average Resource 
Effectiveness . 



Total Number of Attacks of Enemy Targets [] . 



Number of Attacks of Enemy Targets in the Primary (P) 
Attack Axis [] . 



Number of Attacks of Enemy Targets in the Diversionary (D) 
Attack Axis [] . 



Total Leakage [%] . Number of true Enemy targets that 
penetrated the Inner Defense Zone/Total number of threats. 



Total 


Leakage in 


(P) 


[%] • 


Total 


Leakage in 


(D) 


[%] • 



Total Number of Attacks of Decoys [] . 
Total Number of Attacks of Neutrals [] . 



Number of Attacks on Low Threat Targets []. 
Number of Attacks on Medium Threat Targets [] . 



Number of Attacks on High Threat Targets [] . 

2 . Team Strategy Measures 

• Average Resource Effectiveness (AREF) [%] . An enemy 
target requires the allocation of a pre-determined amount 
of resources (weapons) to be attacked with maximum 
effectiveness (i.e., completely destroyed) . The amount of 
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required resources is a function of the damage— producing 
capability of the target. This capability is indicated by 
the second attribute of the target , i.e., its value 
(Low=2-4, Medium=4-6, and High=6-8) . Taking into account 
that each X subplatform contains 1 resource, and each Y 
subplatform contains 2 resources, Table 3 describes the 
resource requirements for attacks. 



TABLE 3 



RESOURCE REQUIREMENTS 



ENEMY TARGET 
CLASS 


LOW 


(2-4) 


MEDIUM (4-6) 


HIGH (6-8) 


RESOURCES 

REQUIRED 


1 


2 


3 


SUBPLATFORMS 

NEEDED 


IX 


or 1Y 


2X or 1Y 


3X or 1X+1Y 



• The resource effectiveness of an attack on an enemy target 
is computed as the square of the ratio of resources 
allocated/resources required. For example, suppose that 
a high-valued enemy has been identified. Three resources 
are required for a complete kill. If a DM allocates IX + 
1Y platforms (three resources) , the resource effectiveness 
(REF) is (3/3) 2 =100%. If a DM allocates only 1Y platform 
(two resources), the REF is (2/3) 2 =44%. There is no bonus 
or penalty for an over-allocation of resources. For 
instance if, in the case described above, the DM allocates 
4X platforms when only three are required, the REF will 
still be 100%. The AREF is computed as the average of all 
REF on attacks on enemy aircraft by the blue team. 

• AREF on Low Threat targets [%] . 



• AREF on Medium Threat Targets [%] . 
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AREF on High Threat Targets [%] . 
Number of Attacks by Platforms [] . 



Average Number of Platforms used for Identification []. 
Averaged across all targets identified. 



Average Team Latency [sec] . Delay between the appearance 
of a target and the very first team action on that target 
(communication, identification, or attack) . When averaged 
across all targets, it represents a measure of the overall 
reaction time of the team. 



Average Identification Delay [sec] . Delay between the 
appearance of a target and the first team identification 
of that target. Averaged across all tasks identified by 
the team. 



Time of First Primary (P) Attack Determination [sec] . 

Time of First Diversionary (D) Attack Determination 
[sec] . 

Number of Attack Determinations []. 



Ratio of Targets Identified as Enemy but not Attacked [%] . 



Number of Direct Attacks by Team Leader [] . 



Average Attack Location [nm] . Distance between an attack 
location and the center of the battle group (A/C) . 
Averaged across all tasks attacked by the team. 



Average Identification Confidence []. 



Average Number of Identification Actions per Task 
Identified [ ] . 



3 . 



Team Coordination Measures 



Total Communication Rate [msg/min] . Total team 

communication traffic average over the length of the 
scenario. (Note: each broadcast counts as three 

messages) . 



Number of Messages per Target Attacked []. 



Number of Platform Requests [] . 

Number of Advices [] . Number of "advise transfer" 
messages issued by the leader to transfer a platform 
between subordinates . 



Vertical Downward Communication []. Number of messages 
issued by the leader to the subordinates. 

Vertical Upward Communications [] . Number of messages 
issued by the subordinates to the leader. 



Horizontal Communication [] . Number of messages issued by 
a subordinate to another subordinate. 



Number of Platform Transfers by Subordinates [] . 



Number of Platform Transfers by the Leader []. 



Number of Wasted Attacks [] . A wasted attack may occur in 
the overlap sectors if a DM attempts to attack a target 
already under attack by another DM. This measure is 
indicative of a lack of planning and attack coordination 
among the team members . 



IV. RESULTS 



A. DATA ANALYSIS 

Due to the large quantity of dependent variables in the 
data set it was necessary to reduce the number to a more 
manageable size. The experiment began with 49 dependent 
variables and through heuristic methods was reduced to 15. 
The original data set contained several dependent variables 
that were subsets of other dependent variables. These subsets 
provided redundant information that was of limited value and 
were eliminated from further consideration. The values of 
several other dependent variables were discovered to depend 
upon the experimental design and were also eliminated. An 
examination of the means of the remaining variables showed 
that the values of several of the variables changed little 
regardless of the independent conditions and were removed from 
consideration. Two variables were not considered because 
their values were affected by mechanical or software error. 

Table 4 shows the dependent variables that remained, what 
they represented, their means, and their standard deviations. 
The data that remained were analyzed using an univariate 
analysis of variance (ANOVA) computed using the General Linear 
Models Procedure of the SAS statistical software program (SAS 
Institute Incorporated, 1988) . 
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TABLE 4 



FINAL DEPENDENT VARIABLE LIST 



NUMBER 


DEFINITION 


MEAN 


ST DEV 


1 


Final Team Strength 


64.67 


** 

CO 

H 

C4 


2 


Number of False Alarms 


5.12 


3.30 


5 


Number of Correct Neutral Identifications 


19.71 


3.27 


6 


Correct Enemy Classification Ratio 


0.80 


0.18 


24 


Average Resource Effectiveness 


0.88 


0.10 


30 


Average Team Latency [sec] 


278.84 


81.72 


32 


Time of First P Attack Determination [sec] 


962.07 


222.42 


38 


Average ID Confidence 




0.26 


39 


Average Number of ID Actions per Task 
Identified 


2.68 


0.82 


40 


Total Communication Rate 


0.81 


0.48 


41 


Number of Messages per Target Attacked 


1.69 


1.36 


44 


Vertical Downward Communication 


9.12 


12.60 


4 5 


Vertical Upward Communications 


4 . 05 


4 . 19 


46 


Horizontal Communications 


10.24 


6.98 


47 


Number of Platform Transfers by Subordinates 


1.95 


1.85 



The ANOVA procedure produces a value, p, which represents 
the probability of making an error in claiming that a 
dependent variable is affected by different levels of an 
independent variable. A value of p < 0.05 is considered 
significant. A value of 0.05 < p < 0.1 is considered 
marginally significant. The ANOVA method also identifies 
significant interactions between two or more independent 
variables. (Miao, Luh and Kleinman, 1990) 
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B . EXPERIMENTAL RESULTS 



Two different approaches were taken in conducting the 
ANOVA. The first approach considers between cell interactions 
(see figure 1) , while the second approach involves between 
team comparisons. A third approach was conducted using 
correlation techniques and an analysis of the processes which 
affected team performance. 

1 . Between Cells Analysis 
a. Summary Statistics 

Summary statistics (means and standard deviations) 
for each of the 15 dependent variables are given in table 5. 
The highest final team strength were attained in situations 
with an active leader and low uncertainty. In situations with 
an active leader and high uncertainty there was a significant 
difference in final team strength between a decentralized 
information structure and a centralized information structure 
(54.00 vs. 63.29) (See figure 4) . Teams in the former 
situation (high uncertainty, decentralized information 
structure, and active leader cell) also showed considerably 
less team latency when the information structure was 
decentralized as opposed to all other cells. Under conditions 
of high uncertainty the first primary attack determination 
took longer when the leader was passive and the information 
structure was decentralized (1063.43 sec) or if the leader 
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TABLE 5 



SUMMARY STATISTICS - BETWEEN CELLS ANALYSIS 



MEANS 

STANDARD DEVIATIONS 









CELLS 




1 




LOW 


LOW 


PASS 


PASS 


■ IGB 


BIGS 




DECENT 


CENT 


DECENT 


CKHT 


DECENT 


CENT 




ACTIVE 


ACTIVE 


aiGB 


Bios 


ACTIVE 


ACTIVE 


1 


73.00 


73.41 


61.88 


65.14 


54.00 


63.29 




24.30 


24.30 


21.90 


20.92 


13.87 


25.4* 


2 


2.29 


3.14 


5. 86 


6.29 


7.14 


6.00 




1.89 


1.35 


2.67 


3.59 


4.80 


2.45 


5 


22.57 


21.71 


19.14 


18.57 


17.86 


1 ■ . 43 




1.90 


1.25 


2.67 


3.46 


4.60 


2.37 


1 


0.14 


7.14 


0.*1 


2.72 


0.77 


0.84 




0.17 


0.88 


0.14 


0.93 


0.21 


0.10 


24 


0 . *9 


2.28 


0.85 


2 .28 


0.87 


0.85 




0.09 


2.0< 


0.13 


2.06 


0.08 


0.11 


30 


2*9.70 


2*9.20 


272.89 


2*9.?* 


2SS.1I 


2*7.98 




95.96 


79.32 


• 6.23 


95.96 


74.58 


95.05 


32 


968.71 


1001.43 


1063.43 


•48.14 


8*7.29 


1003.43 




133.23 


209.14 


261.91 


288.95 


229.21 


1*5.57 


44 


2.15 


2.13 


1 .82 


• . 8 4 


1.77 


1.95 




0.22 


0.17 


0.26 


0.17 


0.23 


0.78 


44 


2.55 


2.5* 


2.19 


2.72 


2.57 


2.76 




0 . «1 


0.85 


0.87 


0.93 


0.92 


0 . *5 


44 


0.71 


0. *6 


0.57 


0.84 


0.62 


0.83 




0.38 


0.63 


0.52 


0.17 


0.65 


0.37 


44 


1.68 


2.28 


1.93 


0.93 


1.40 


1.95 




0.75 


2.06 


2.07 


0.57 


1.15 


0.7* 


44 


11.14 


11.57 


7.43 


3.00 


10.29 


2*7.9* 




9.60 


14.23 


15.89 


4.36 


17.75 


95.85 


45 


3.37 


5.71 


2. 86 


3. 14 


3. *6 


5.14 




2 .*2 


5. *8 


3.34 


2.91 


4.67 


5. 34 


46 


10.14 


12.86 


9.57 


8.57 


7.57 


12.71 




6.74 


10.35 


5.74 


2.70 


5, 44 


9.14 



was active and the information structure was centralized 
(1003.43 sec) as opposed to a passive leader with a 
centralized information structure (848.14 sec) or an active 
leader with decentralized information structure (887.29 
sec) (See figure 5) . Under high uncertainty, with an active 
leader more identification actions were taken per task with 
centralized information than with decentralized information; 
however, with a passive leader more identification actions 
were taken per task with decentralized information than with 
centralized information (See figure 6) . With a passive leader 
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FINAL TEAM STRENGTH 




Figure 4 - Final Team Strength 

and high uncertainty there were significantly more vertical 
communications in decentralized (7.43) as opposed to 
centralized (3.00) information structures. With an active 
leader there were more horizontal communications with 
centralized information than with decentralized information; 
however, with a passive leader there were more horizontal 
communications with decentralized information than with 
centralized information (See figure 7) . 

Jb. Analysis of Variance 

Table 6 summarizes the results of the univariate 
analysis of variance (ANOVA) for the selected dependent 
variables. There was a difference in the number of false 
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TIME OF FIRST P ATTACK DETERMINATION 




PASSIVE -+“ ACTIVE 

Figure 5 — Time of First Primary Attack Determination 

alarms found in each of the cells at the 0.0234 significance 
level. This shows that in low uncertainty situations the 
number of false alarms is markedly less than in high 
uncertainty situations. Supporting the same hypothesis is the 
fact that the number of correct neutral identifications was 
significant at 0.0196 significance level. This shows that in 
low uncertainty situations teams were better able to identify 
neutral tasks than in high uncertainty situations regardless 
of the information structure or leader role. Average 
identification confidence was found to be significant at the 
0.0020 level. This shows that in low uncertainty situations 
teams had higher average identification confidence than in 
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AVERAGE ID ACTIONS PER TASK 




high uncertainty situations. The dependent variables which 
showed significance in the between cells analysis all measured 
an impact of uncertainty. Dependent variables measuring 
impacts of the other independent variables did not show 
significance as measured by the ANOVA' s . This supports the 
hypothesis that only uncertainty has a significantly different 
impact between the cells . 

c. Team Composition Summary 

Table 7 shows the composition of each team 
participating in the experiment. This table shows time— in- 
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HORIZONTAL COMMUNICATION 




— — PAS8IVE — t— ACTIVE 

Figure 7 - Horizontal Communication 

service, branch of service, specialty, team average time-in- 
service, tactical rating, and strategy rating. Tactical 
rating is a percentage of the team members that are trained to 
serve in tactical positions. This includes Surface Warfare 
Officer (SWO) , Naval Flight Officer (NFO) , Air Defense 
Artillery Officer (ADA) , Pilot/Electronic Warfare Officer 
(EW) , Anti-Air Warfare Officer (AAW) , and Infantry Officer 
(INF) . The non— tactical positions included Computer Related 
Specialty Officer (COMP) , Communications Officer (COMMS) , 
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TABLE 6 



ANOVA - BETWEEN CELLS ANALYSIS 



DEPENDENT 

VARIABLE 


SUM OF SQUARES 


F VALUE 


Pr > F 


1 


1608.76 


0.65 


0.6668 


2 


130.97 


2.99 


0.0234 


5 


132.28 


3.11 


0.0196 


6 


0.03 


0.26 


0.9309 


24 


0.02 


0.42 


0.8332 


4*1 


6333.03 


0.17 


0.9719 


3 « 


225045.93 


0. 65 


0.0026 


38 


1.09 


4.73 


0.0020 


39 


a . e 9 


0.47 


0.9733 


40 


0.98 


0.83 


0.5382 


41 


7.98 


0.84 


0.5271 


44 


395.26 


0.47 


0.7990 


45 


45.33 


0 . 48 


0.7860 


46 


163.33 


0.64 


0.6703 



Instructor (INST ) , and Public Affairs Officer (PAO) . The 
strategy rating information was gathered through team 
debriefings following their last experimental trial. Each 
team was asked to describe their strategy for handling overlap 
areas, transferring resources, and assigning confidence levels 
to identified tasks. Some of the teams had a coherent 
strategy while others did not. 

d. Summary Statistics 

Summary statistics (means and standard deviations) 
for each of the 15 dependent variables are given in table 8 . 
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TABLE 7 

TEAM COMPOSITION 



TEAM 


DM # 


TIME 


BRANCH 


PRIMARY 

TRAINING 


AVG TIME 


TACTICAL 


STRATEG 

Y 


A 


0 


6.5 


USN 


SWO 


10.6 


50% 


YES 


1 


6 


USN 


SWO 


2 


13 


USAF 


COMP 


3 


17 


USAF 


COMP 


B 


0 


8 


USAF 


COMMS 


10.3 


50% 


NO 


1 


12 


USN 


NFO 


2 


5 


USN 


SWO 


3 


16 


USAF 


COMMS 


C 


0 


5.75 


USMC 


ADA 


6.1 


75% 


YES 


1 


6 


USN 


EW 


2 


5 


USAF 


COMP 


3 


7.5 


USN 


AAW 


D 


0 


7 


USN 


INST 


9.0 


0% 


NO 


1 


16 


USAF 


COMMS 


2 


7 


USAF 


COMMS 


3 


6 


USAF 


COMMS 


E 


0 


11 


USN 


NFO /AAW 


10.4 


100% 


YES 


1 


16 


USMC 


INF 


2 


6.5 


USN 


SWO 


3 


8 


USA 


INF 


F 


0 


6 


USN 


SWO 


6.75 


75% 


YES 


1 


7 


USAF 


COMMS 


2 


6 


USN 


SWO 


3 


8 


USN 


SWO 


G 


0 


6 


NSA 


COMMS 


6.1 


50% 


NO 


1 


4 


USN 


SWO 


2 


5.5 


USN 


SWO 


3 


9 


USN 


PAO 



The three teams with high tactical ratings (75%+) showed 
significantly higher final team strength, higher correct enemy 
classification ratios, and better average resource 

effectiveness . The two teams with the lowest final team 

strength (A and D) had poor resource effectiveness, lower 
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average identification confidence levels, and significantly 
more vertical upward communications. 

e. Analysis of Variance 

Table 9 summarizes the results of the univariate 
analysis of variance (ANOVA) for the selected dependent 
variables. In the between teams analysis all of the selected 
dependant variable ANOVA' s showed a significance level less 
than 0.05. This shows that significant differences existed 
between each of the teams' performance, strategy, and 
communication methods. 

2 . Process Analysis 

As was stated before in Chapter III the dependent 
variables were broken into three categories; Measures of 
Performance, Measures of Strategy, and Measures of 
Communication. Final Team Strength, Number of False Alarms, 
Number of Correct Neutral Identifications, Correct Enemy 
Classification Ratio, and Average Resource Effectiveness are 
all measures of performance. Average Team Latency, Time of 
First Primary Attack Determination, Average Identification 
Confidence, and Average number of ID Actions per Task 
Identified are all measures of strategy. Total Communication 
Rate, Number of Messages per Target Attacked, Vertical 
Downward Communication, Vertical Upward Communication, and 
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TABLE 8 



SUMMARY STATISTICS - BETWEEN TEAMS ANALYSIS 

MEANS 

STANDARD DEVIATIONS 



TEAMS 


DVAR 


A 


B 


C 


D 


E 


F 


G 


















1 


44.33 


61.33 


85.00 


41.17 


71.33 


80.67 


68.83 | 




22.07 


26.81 


22.75 


19.46 


14.05 


6.95 


17.11 


2 


4.83 


4.67 


3.33 


4.50 


4.17 


5.00 


9.33 




2.94 


3.20 


1.76 


1.17 


5.79 


3.00 


2.35 


C 


19.83 


20.17 


21.50 


20.50 


20.83 


19.83 


15.33 




2 . 94 


3.15 


1.76 


0.82 


5.58 


3.00 


2.46 


6 


0. 84 


0.76 


0.92 


0.67 


0.94 


0.93 


0.57 




0.06 


0.13 


0.17 


0.06 


0.23 


0.08 


0.17 


24 


0.80 


0.86 


0.98 


0.78 


0.90 


0.96 


0.85 




0.13 


0.11 


0.07 


0.07 


0.06 


0.06 


0.08 


30 


375.03 


338 . 02 


260.29 


303.25 


131.28 


307.56 


236.42 




59.10 


25.58 


98.21 


110.48 


45.68 


60.49 


71.60 


32 


879.83 


951 . 17 


836.50 


708.83 


956.33 


1201.33 


1200.50 




229.74 


141.47 


119.88 


176.31 


83.55 


167.12 


219.25 


38 


1.64 


1 . 97 


2.03 


1.73 


2.11 


1.96 


1 . 98 




0.27 


0.13 


0.26 


0.14 


0.21 


0.15 


0.14 


39 


1.66 


1.54 


2.92 


2.36 


3.78 


3.43 


3.07 




0.70 


0.30 


0.67 


0.38 


0.38 


0.87 


0.54 


30 


1.25 


0.39 


0.65 


0.71 


0.90 


0.18 


0.60 




0.24 


0.28 


0.50 


0.61 


0.39 


0.14 


0.27 


41 


3.04 


1.10 


1.38 


1.53 


2.51 


0.49 


1.80 




0.69 


0.51 


1.28 


2.28 


1.21 


0.96 


0.73 


30 


26.33 


1.83 


4.17 


7.33 


20.50 


0.33 


3.33 




1.51 


3.33 


12.83 


17.95 


3.02 


12.20 


7.88 


45 


9.33 


2.33 


4.83 


7.33 


3.50 


0.67 


0.33 




3.10 


4.59 


3. 67 


2.10 


0.84 


4.42 


2.93 


46 


8.00 


9.33 


13.67 


10.17 


7.67 


5.33 


17.50 




6.57 


8.33 


7.82 


2.40 


11.62 


6.31 


4.83 



Horizontal Communication are all measures of communication. 
A correlation analysis was performed using the correlation 
procedures in the SAS statistical analysis software which 
produced a correlation matrix showing interactions between the 
variables (SAS Institute Incorporated, 1988) . 
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TABLE 9 



ANOVA - BETWEEN TEAMS ANALYSIS 



DVAR 


SUM OF SQUARES 


F VALUE 


Pr > F 


1 


10248.33 


6.42 


0.0146 


2 


135.23 


2.54 


0.0382 


5 


146.90 


2.54 


0.0146 


6 


4.ii 


16.67 


0.0146 


24 


0.20 


7.15 


0.0001 


30 


228545.90 


29.44 


V# 

O 

o 


32 


1205442.62 


8.55 


0.0001 


38 


1.03 


3.45 


0.0146 


39 


26.45 


107.54 


0.0001 


40 


4.29 


4.81 


0.0011 


41 


26.55 


3.13 


0.0146 


44 


3703.90 


7.72 


0.0001 


45 


406. 90 


7 . 58 


0 . 0001 


46 


605. 95 


2.54 


0.0382 



An analysis of the correlation matrix showed a 
moderate negative correlation (-0.35, P< 0.0217) showing that 
increased Vertical Upward Communications leads to decreased 
Average ID Confidence. Furthermore, there exists a moderate 
correlation (0.42, P<0.0057) indicating that decreased Average 
ID Confidence resulted in reduced Average Resource 
Effectiveness. Additionally, there is a strong correlation 
(0.82, P<0.0001) demonstrating that reduced Average Resource 
Effectiveness caused a lower Final Team Strength. This 
analysis is supported by the summary statistics of the between 
teams analysis. 
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Further analysis of the correlation matrix 
demonstrated that a moderate correlation (0.36, PC0.0197) 
existed showing that an increase in the Average Number of ID 
Actions per Task Identified led to an increase in the Average 
ID Confidence level. Also, a there was a moderate correlation 
(0.42, P<0.0057) indicating that an increase in the Average ID 
Confidence level caused an increase in the Average Resource 
Effectiveness. As indicated before increased Average Resource 
Effectiveness results in higher Final Team Strength. This 
analysis is also supported by the summary statistics of the 
between teams analysis. 

A moderate negative correlation (-0.41, PC0.0073) 
indicates a decrease in the Vertical Upward Communication led 
to an increase in the Average Number of ID Actions per Task 
Identified. A further moderate correlation exists (0.45, 
PC0.0027) demonstrating that an increase in the Average Number 
of ID Actions per Task Identified led to an increase in the 
Final Team Strength. 
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V. CONCLUSIONS AND FUTURE RESEARCH 



A. GOAL 

The goal of this study was to determine the impact of (1) 
Neutral/Enemy discriminability uncertainty, (2) Team 
information structure, and (3) Leader's involvement in 
resource coordination upon four person teams consisting of 
military decisionmakers. 

B. CONCLUSIONS OF STUDY 

Three approaches were taken in the analysis of the 
experimental results; between cells analysis, between teams 
analysis, and an analysis of the processes that affected team 
performance. The between cells analysis showed that of the 
three independent variables the level of uncertainty had the 
most significant impact on team performance. With an active 
leader there was an increase in Horizontal Communications, 
Average ID Actions per Task, and Time of First Primary Attack 
Determination when moving from a decentralized information 
structure to a centralized information structure. The 
opposite was true when the leader was passive. These 
variables showed a decrease when moving from a decentralized 
information structure to a centralized information structure. 
The ANOVA only showed three dependent variables to be 
significant; Number of False Alarms, Number of Correct Neutral 
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Identifications, and the Average ID Confidence. The first two 
of these are very closely related while the third is closely 
related to the level of uncertainty. 

The between teams analysis shows that the level of 
tactical training within a team directly influenced the 
performance of that team. Teams with a more tactical 
background developed a more coherent team strategy. Average 
time-in-service appeared to have little or no impact on team 
performance. The two teams with the lowest Final Team 
Strength demonstrated poor resource effectiveness, lower 
confidence levels, and significantly more upward 
communication . 

The process analysis demonstrated an association between 
communication methods, team strategy, and team performance. 
There were more significant correlations between strategy 
measures and performance measures than communication measures 
and performance measures, although the communication measures 
were highly correlated to each other. This analysis showed 
that the subordinates who tried to pass a large amount of 
information up the chain of command tended to assign that 
information much lower confidence levels causing less 
effective use of resources by the team and subsequently lower 
team strength. Conversely, the subordinates who passed less 
information up the chain-of-command were able to take more 
sensor readings and attained a higher team strength. 
Additionally, teams which took multiple sensor readings on 
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unidentified targets were able to assign a higher confidence 
level to each task leading to better resource effectiveness 
and finally a higher final team strength. 

C. LIMITATIONS AND FURTHER RESEARCH 

There were five primary limitations in the conduct of this 
experiment. First, communications were severely limited. 
Each decisionmaker was limited to a pre-established set of 
messages which was very restrictive. A recommendation would 
be to include either plain-text messages that can be entered 
from the keyboard or to establish voice communications. 

Second, the workspace was very confined and the 
decisionmakers were not adequately separated from each other. 
A recommendation would be to ensure physical separation of all 
decisionmakers . 

Third, during the conduct of the experiment it was 
discovered that sometimes the leader was unable to engage 
targets due to a problem in the software. After several 
discussions with the primary programmer it was decided that 
the software repair would have to wait until after the 
conclusion of this experiment. This led to removing one of 
the dependent variables from the data set (DVAR 36 - Number of 
Direct Attacks by Team Leader) . 

Fourth, the hardware that was used for this experiment 
included monochrome monitors . The use of color monitors would 
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enhance the decisionmakers ability to decipher information 
from the screen. 

Fifth, decisionmakers were unable to filter the 
information presented on the screen. The ability to look at 
only targets identified as enemy or neutral would reduce 
clutter and increase readability leading to less confusion. 
This is especially true for the team leader. 

Several follow-on studies are suggested by the results 
provided here. First, a study of the impact of implicit 
coordination, as a strategy, on communications and team 
performance . In the RAINCOAT experiment it became evident 
that some of the teams developed very comprehensive team 
strategies. They developed rules for assigning confidence 
levels to task identifications, resource allocation in the 
overlap areas and, types of communication methods. The impact 
of these rules showed up in the RAINCOAT experiment as 
improved team performance. Future studies could concentrate 
on analyzing this implicit coordination as an independent 
variable in order to quantify its impact. 

Another enhancement to this study would be to move the 
decisionmakers into a more tactical setting and introduce 
external noise as an added pressure. The experiment was 
conducted in a quiet office setting. The decisionmakers were 
isolated from external interference and even prevented from 
talking to each other. Future studies could focus on 
introducing noise into the environment in order to analyze its 
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impact on the decisionmaking process- This would move the 
paradigm closer to a real life setting. 

Third, subjective workload assessment data was gathered 
for this experiment. An analysis should be done to determine 
the impact of perceived workload on team performance. Data 
exists for each decisionmaker and for each team. 
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